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Summary  Page 


THE  PROBLEM 

To  determine  whether  NSMRL  computer  program  to  diagnose  chest 
pain  is  suitable  for  use  by  corpsmen  at  sea. 


THE  FINDINGS 

132  patients  with  confirmed  diagnoses  of  chest  pain 
(myocardial  infarction,  angina,  chest  infection,  or  other) 
presenting  at  Naval  Hospitals  at  Portsmouth,  Charleston,  and  San 
Diego  were  diagnosed  by  the  emergency  room  physicians  and  the  NSMRL 
computer  program.  The  percentage  of  correct  diagnoses  by  the 
computer  (72%)  was  not  significantly  different  from  the  percentage 
of  correct  diagnoses  by  the  ER  physicians  (79%) .  The  physicians 
were  significantly  better  than  the  computer  program  at 
distinguishing  myocardial  infarction  from  angina.  However,  the 
computer  program  was  equally  successful  at  determining  whether  a 
patient  was  suffering  from  myocardial  infarction. 


APPLICATION 

The  computer  program  for  diagnosing  chest  pain  should  be  of 
assistance  to  the  independent  duty  corpsman  in  the  diagnosis  and 
management  of  patients  aboard  submarines  presenting  with  chest 
pain. 


Administrative  Information 

This  investigation  was  conducted  under  Naval  Medical  Research 
and  Development  Command  Research  Work  Unit  63706N  -  M0095 . 005-5010, 
"Sea  trials  for  computer-based  medical  diagnostic/patient 
management  system  for  use  aboard  SSN/SSBN  submarines."  It  was 
submitted  for  review  on  25  July  1990,  approved  for  publication  on 
7  December  1990,  and  has  been  designated  as  Nav:il  Submarine  Medical 
Research  Laboratory  Report  No.  1162. 


Abstract 


We  evaluated  the  performance  of  a  computer  program  designed  to 
assist  corpsmen  in  managing  patients  who  present  at  sea  with  chest 
pain.  The  diagnostic  accuracy  of  the  program  for  three  common  and 
serious  causes  of  chest  pain  —  myocardial  infarction  (MI) ,  angina, 
chest  infection  —  as  well  as  its  accuracy  for  nonspecific  chest 
pain,  was  compared  with  that  of  emergency  room  physicians  at  three 
Naval  hospitals  for  132  patients.  The  percentage  of  correct 
diagnoses  by  the  computer  program  (72%)  was  not  significantly 
different  from  that  of  the  physicians  (79%) ,  although  the  accuracy 
of  the  physicians  was  better.  The  computer  program  was  able  to 
distinguish  MI  from  the  other  diseases  as  well  as  the  physicians 
(i.e.  was  as  sensitive,  83%  vs.  84%)  and  at  the  same  time  made 
fewer  false  diagnoses  of  MI  (i.e.  was  more  specific)  .  The 
computer's  ability  to  distinguish  among  cardiac  illnesses  was  not 
as  good  as  that  of  the  physicians  and  it  misdiagnosed  more  cases  of 
angina.  The  computer  program  is  suitable  for  use  by  medical 
personnel  practicing  in  isolated  locations  and  may  help  improve 
diagnostic  accuracy  in  cases  of  MI.  Its  failures  emphasize  that  it 
cannot  substitute  for  medical  personnel,  but  can  be  helpful  to 
medical  personnel  when  a  patient's  diagnosis  is  uncertain. 


Introduction 


There  has  in  recent  years  been  much  interest  in  using  the 
computer  as  an  aid  in  medical  diagnosis  (Rogers,  Ryack,  and 
Moeller,  1979) .  This  interest  is  greatest  in  the  submarine  fleet. 
Physicians  are  assigned  only  to  newly  commissioned  Trident 
submarines;  complete  responsibility  for  medical  care  otherwise 
rests  with  the  ship's  independent  duty  hospital  corpsman.  With 
neither  the  training  nor  experience  of  a  practicing  physician,  the 
corpsman  must  decide  whether  or  not  to  recommend  a  medical 
evacuation.  To  ensure  that  his  patients  receive  all  needed  care, 
particularly  when  a  patient's  condition  is  puzzling,  corpsmen  find 
evacuation  to  a  physician's  care  an  attractive  option.  But 
evacuations  are  costly,  dangerous  to  both  the  patient  and  the  deck 
crew,  and  operationally  undesirable.  Reducing  unnecessary 
evacuations  is  a  desirable  goal.  Submariners,  therefore,  would 
find  it  useful  to  have  a  tool  with  a  demonstrated  ability  to  assist 
the  corpsman  in  making  a  good  decision  and  reducing  the  number  of 
inappropriate  —  that  is,  premature  or  delayed  —  medical 
evacuations. 

NSMRL  has  developed  microcomputer-based  medical  support 
systems  to  aid  the  corpsmen  in  treating  a  variety  of  maladies. 
Some  of  the  systems  are  based  on  one  devised  by  de  Dombal  (1973; 
1979) ,  but  they  have  been  specifically  tailored  for  the  submarine 
population  (Osborne,  1984;  Ryack,  1987).  One  provides  information 
to  aid  the  corpsmen  in  the  diagnosis  and  management  of  chest  pain 
(Southerland  and  Fisherkeller ,  1987;  Fisherkeller ,  Southerland,  and 
Moeller,  1987) .  The  corpsman,  using  a  data  collection  form  to 
guide  his  examination,  records  findings  for  27  history  and  20 
physical  exam  categories.  The  program  then  provides  the  user  with 
probabilities  that  the  patient  is  suffering  from  each  of  four 
illnesses:  myocardial  infarction,  angina,  chest  infection,  and 
non-specific  chest  pain. 

Although  the  program  produces  the  probabilities  for  four 
separate  diagnoses,  it  can  be  argued  that  it  is  more  important  for 
the  corpsman  to  determine  which  patients  are  suffering  from  an  MI 
(and  must  receive  specialized  care)  than  it  is  to  distinguish  among 
various  other  medical  diagnoses.  The  reason  is  that  the  corpsman 
is  trying  to  decide  whether  or  not  a  patient  must  be  evacuated.  It 
would  be  most  useful  if  the  program  would  help  to  identify  patients 
with  MI. 

Although  a  new  program  may  appear  to  be  similar  to  another 
which  has  been  validated  (de  Dombal,  1973) ,  it  still  requires 
independent  validation.  There  have  been  previous  field  trials  of 
such  programs  in  the  Navy  (Henderson,  et  al.,  1981),  but  these  have 
sought  only  to  determine  how  easily  the  corpsmen  can  use  the 
program  and  how  well  they  accept  it;  they  have  not  attempted  to 
validate  the  programs  themselves. 
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This  prospective  validation  of  the  chest  pain  program  was 
begun  to  assess  its  reliability  and  consequent  usefulness  to 
independent  duty  corpsmen.  This  report  compares  the  diagnostic 
accuracy  of  the  NSMRL  chest  pain  program  with  that  of  emergency 
room  physicians  for  patients  who  were  reasonably  representative  of 
an  active  duty  Navy  population. 

METHOD 


Subjects 

The  subjects  were  132  male  or  female  active  duty  or  retired 
Navy  personnel  or  their  dependents  between  the  ages  of  15  and  50. 
They  presented  with  chest  pain  in  the  emergency  rooms  of  the  Naval 
Hospitals  at  Portsmouth,  VA,  Charleston,  SC,  and  San  Diego,  CA, 
from  May  31,  1988  to  September  30,  1988  and  volunteered  to 
participate  in  the  study.  The  patients  also  met  the  requirement 
that  they  not  have  a  prior  history  of  myocardial  infarction, 
angina,  or  such  chronic  illness  as  diabetes,  since  these  conditions 
are  grounds  for  exclusion  from  submarine  service. 


Technicians 

Six  research  technicians  —  two  at  each  hospital  —  collected 
the  data.  They  were  trained  in  the  use  of  the  chest  pain  data 
sheet  by  a  Navy  medical  officer  over  a  two-day  period.  The 
technicians  worked  consecutive  8-hour  shifts  between  8:00  a.m.  and 
midnight,  five  days  a  week,  for  the  four  month  period. 

Procedure 

Patients  who  presented  at  the  emergency  room  with  a  complaint 
of  chest  pain  were  interviewed  by  a  research  technician  who 
described  the  study  and  solicited  their  participation.  Patients 
who  agreed  to  participate  in  the  study  signed  consent  forms  and 
then  the  technician  recorded  case  history  details  Wwile  waiting  for 
the  examining  physician.  This  included  such  things  as  the 
patient's  sex  and  age,  the  history  of  smokinc ,  previous  surgery, 
history  of  MI,  angina,  bronchitis,  previous  illnesses,  etc.  While 
the  patient  was  being  examined  by  the  physician,  the  technician 
listened  to  the  examination  and  recorded  the  answers  to  the 
questions  asked  by  the  computer  program  on  the  data  sheet  shown  in 
Figure  1.  For  example,  the  chest  pain  program  asks  the  site  of  the 
pain,  the  presence  of  radiation,  duration  and  severity  of  the  pain, 
and  so  on.  The  required  information  could  be  obtained  by  watching 
the  examination.  The  physician  was  asked  to  verbalize  his  findings 
as  he  conducted  the  examination  so  that  the  technician,  listening, 
could  complete  the  data  sheet.  Sometimes  the  physician  was  too 
busy  to  provide  a  complete  report;  in  these  cases  the  technician 
obtained  the  missing  information  from  the  patient's  emergency 
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treatment  record.  The  technician  was  not  allowed  to  interview  the 
patient  without  the  permission  of  the  physician  and  was  asked  to 
leave  the  examining  room  if  the  patient  was  uncomfortable.  The 
collected  information  was  later  entered  into  the  computer  program 
to  obtain  the  computer's  diagnosis. 

The  computer  program  for  chest  pain  was  written  to  assign  the 
probabilities  that  a  given  patient's  symptoms  indicated  angina, 
chest  infection,  myocardial  infarction,  and  non-specific  chest 
pain.  The  category  "Angina"  included  both  stable  and  unstable 
angina.  Chest  infection  was  defined  as  any  infection  of  the 
pleural  cavity  (including  pleuritis  and  pericorditis)  which  would 
require  medical  intervention,  such  as  pneumonia  or  bronchitis. 
Cases  of  pneumothorax  are  also  included  in  this  chest  pain  category 
because  they  exhibit  symptoms  of  pleuritic  pain.  Non-specific 
chest  pain  was  defined  as  chest  pain  that  is  not  serious,  not  life- 
threatening,  and  does  not  require  urgent  medical  intervention. 
Mechanical  chest  discomfort  and  GI  tract  pain  fall  in  this 
category. 

The  computer  diagnoses  were  compared  with  diagnoses  from  two 
other  sources.  The  first  was  the  diagnosis  arrived  at  by  the 
examining  physician  in  the  emergency  room.  If  the  patient  was 
admitted  to  the  hospital,  there  would,  of  course,  be  further 
observations  and  tests  leading  to  a  "discharge  diagnosis"  entered 
in  the  patient's  records.  The  discharge  diagnosis  was  presumably 
more  accurate  and  was  accepted  as  the  "final"  diagnosis.  If  a 
patient  was  not  admitted,  there  was  a  follow-up  telephone  call  to 
the  patient  by  a  physician  or  nurse  at  least  three  weeks  after  the 
visit  to  the  emergency  room  to  determine  if  subsequent  events  cast 
doubt  on  the  emergency  room  discharge  diagnosis.  If  a  subject  had 
had  further  medical  care  related  to  the  same  illness,  the  results 
of  that  care  were  considered  in  establishing  a  "final"  diagnosis. 

In  comparing  the  diagnosis  reached  by  the  computer  program 
with  those  reached  by  physicians  after  evaluating  a  patient  during 
an  ER  visit  or  a  hospitalization,  it  is  important  to  remember  that 
the  computer  program  is  designed  for  use  in  isolated  situations 
where  X-rays,  electrocardiograms  (EKG) ,  laboratory  procedures  such 
as  creatine  phosphokinase  (CPK)  and  serum  glutamate  oxaloacetic 
acid  transferase  (SGOT)  assays  and  consultations  with  specialists 
are  generally  not  available.  The  diagnosis  of  the  computer  program 
must,  therefore,  be  based  on  only  a  fraction  of  the  information  to 
which  the  ER  physician  has  access. 

RESULTS 

Table  1  compares  the  final  diagnoses  with  the  diagnoses  made 
by  the  emergency  room  physicians  and  with  those  made  by  the 
computer  program.  There  are  132  computer  diagnoses  but  only  130 
emergency  room  diagnoses  because  two  of  the  emergency  treatment 
records  were  not  available. 
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Table  1  shows,  for  example,  that  of  the  132  cases,  9  were 
finally  diagnosed  as  MI,  6  as  angina,  9  as  chest  infection,  and  108 
as  cases  of  non-specific  chest  pain.  Overall,  79%  of  the  diagnoses 
made  by  the  emergency  room  physicians  and  72%  of  the  computer 
diagnoses  agreed  with  the  final  diagnoses.  These  percentages  are 
not  significantly  different  (t  =  1.35,  df  =  129,  p  <  .50). 


Table  1.  Comparison  of  the  diagnoses  by  the  computer  program  and 
the  emergency  room  physicians  with  the  final  diagnoses  for 
myocardial  infarction  (MI) ,  angina  (AN) ,  chest  infection  (Cl) ,  and 
non-specific  chest  pain  (NS) . 
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Of  the  9  cases  of  MI,  the  computer  program  accurately 
diagnosed  3/  it  misdiagnosed  5  as  angina  and  one  as  non-specific 
chest  pain.  Each  computer  diagnosis  of  MI  was  confirmed  on 
admission.  Of  8  cases  of  MI,  the  physicians  accurately  diagnosed 
6;  they  misdiagnosed  one  as  angina  and  one  as  non-specific  chest 
pain.  (We  did  not  have  the  physicians*  emergency  room  diagnoses 
for  the  remaining  MI  patient.)  Seventeen  physician  diagnoses  of  MI 
were  not  confirmed  on  admission;  15  patients  were  diagnosed  with 
non-specific  chest  pain  and  2  with  angina. 


The  set  of  diagnoses  established  by  the  emergency  room 
physicians  was  not  significantly  different  from  the  final 
diagnoses,  according  to  a  Chi  Square  test  (X^  =  16.95,  df  =  3,  p  < 
.  001)  . 

Cardiac  Event  vs.  Myocardial  Infarction 

Among  patients  with  MI,  the  computer  program  did  not 
accurately  discriminate  between  MI  and  angina.  It  was 
predominantly  this  factor  and  several  misdiagnoses  of  angina  which 
produced  the  difference  in  the  Chi  Square  results  for  the  computer 
and  physicians. 


However,  the  program  correctly  identified  all  but  one  of  the 
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MI  patients  as  having  either  an  MI  or  angina.  If  we  combine  MI  and 
angina  as  a  diagnosis  of  "cardiac  event,”  the  performance  of  the 
computer  program  and  the  ER  physicians  is  much  more  similar  (Table 
2) .  Both  the  computer  program  and  the  ER  physicians  correctly 
identified  all  but  one  of  the  patients  with  MI;  the  computer 
correctly  identified  106  of  the  123  patients  who  did  not  have  MI, 
and  the  physicians  correctly  identified  96  of  them  (the  emergency 
room  diagnosis  of  one  of  the  non-MI  patients  is  also  missing)  .  The 
Chi  Square  test  now  shows  that  the  computer  program  diagnoses  were 
not  significantly  different  from  the  final  diagnoses  (X^=  2.46,  df 
=  1,  p  <  .15),  whereas  the  ER  diagnoses  were  significantly 
different  from  the  final  diagnoses  (X^  =  5.67,  df  =  1,  p  <  .02). 


Table  2 .  Comparison  of  computer  program  and  emergency  room 
physicians  for  final  diagnosis  of  myocardial  infarction  (MI)  when 
initial  diagnosis  of  cardiac  event  (CE,  i.e.  MI  or  angina)  is  made. 

Computer  Physicians 

Final  Diagnosis  Final  Diagnosis 

MI  Not  MI  MI  Not  MI 

CE  8  17  CE  7  26 

Not  CE  1  106  Not  CE  1  96 


6 


DISCUSSION 


Physicians  evaluating  a  patient  with  chest  pain  err  on  the 
side  of  conservatism  and  admit  to  hospitals,  often  with  a  "rule  out 
MI"  diagnosis,  three  or  more  patients  who  actually  turn  out  to  have 
non-cardiac  causes  of  pain  for  every  patient  who  turns  out  to  have 
suffered  a  heart  attack.  This  approach  is  entirely  appropriate  as 
the  risk  of  imminent  death  associated  with  a  missed  diagnosis  of  MI 
is  far  less  acceptable  than  the  inconvenience  of  a  period  of 
observation  necessary  to  ensure  that  no  heart  damage  has  occurred. 

The  "bias"  toward  overdiagnosis  of  MI  is  magnified  when  the 
person  providing  medical  care  has  less  experience  with  MI  patients 
as  occurs  when  hospital  corpsmen  are  caring  for  ship's  crews  alone 
at  sea.  In  this  context,  a  computer  program  which  is  about  as 
reliable  as  an  emergency  rooii.  '''sician  in  detecting  cases  of  MI, 
but  which  does  not  miss  cases  mo^a  frequently  than  physicians  did, 
might  be  a  useful  tool  to  corpsmen  at  sea. 

In  terms  of  the  overall  percentage  of  correct  diagnoses,  the 
performance  of  the  computer  program  was  not  significantly  different 
from  that  of  the  emergency  room  physicians.  This  finding  rests 
largely  on  the  overwhelming  percentage  of  non-specific  chest  pain 
diagnoses  in  the  sample  populations.  The  physicians  were  better  at 
distinguishing  among  specific  illnesses.  The  program  did  not  do 
well  at  diagnosing  angina,  for  example. 

But  in  evaluating  the  performance  of  the  computer  program 
designed  for  use  on  submarines  and  in  comparing  its  accuracy  with 
that  of  physicians,  it  is  necessary  to  remember  that  the  physicians 
had  access  to  a  variety  of  other  diagnostic  information  not 
available  to  the  program:  detailed  interpretation  of  EKGs,  CPK 
assays,  chest  X-rays,  and  consultations  with  specialists.  It  is 
also  necessary  to  remember  that  the  primary  purpose  of  the  computer 
program  is  to  help  the  independent  duty  corpsman  faced  with  a 
puzzling  case  to  distinguish  between  chest  pain  stemming  from  an  MI 
and  other  causes  of  chest  pain.  The  program's  ability  to  specify 
precisely  what  illness  the  patient  actually  has  is  .lot  as  critical 
as  its  ability  to  correctly  categorize  the  illness  as  an  MI  or  some 
other  condition.  In  this  restricted  sense,  th*"  performance  of  the 
program  is  about  the  same  as  that  of  the  ER  pnysicians. 

Other  computer  programs  have  been  reported  to  diagnose  MI 
quite  accurately.  Goldman  and  his  colleagues  (1982,  1988)  have 
reported  that  their  program  was  significantly  more  specific  than 
were  ER  physicians.  In  an  earlier  study  (Goldman,  Weinberg, 
Weisberg,  et  al.,  1982),  they  identified  nine  critical  questions 
which  bear  on  the  presence  or  absence  of  MI.  One  of  them,  however, 
is  not  appropriate  to  submariners;  it  is  whether  or  not  this  same 
pain  was  previously  diagnosed  as  MI.  If  this  had  been  the  case, 
the  man  would  not  be  serving  on  submarines.  Further,  it  is 
important  to  note  that  in  their  recent  report  (Goldman,  Cook, 
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Brand,  et  al.,  1988)  we  see  that  the  information  requested  by  their 
very  first  question  —  "ST  elevation  or  Q  waves  in  2  or  more  leads, 
not  known  to  be  old?"  —  is  not  available  on  submarines. 
Independent  duty  corpsmen  are  not  routinely  trained  to  make 
detailed  evaluations  of  EKGs,  and  most  do  not  have  EKG  equipment 
available.  Similarly,  another  question  in  Goldman's  algorithm  asks 
about  EKG  changes  associated  with  ischemia  or  strain.  This 
information,  too,  is  not  available  to  most  independent  duty 
corpsmen.  Thus,  a  program  which  may  work  very  well  in  a  typical 
hospital  may  be  ineffective  at  sea.’ 

We  hasten  to  emphasize  that  this  study  shows  that  the  computer 
program  makes  errors.  It  erroneously  categorizes  the  source  of  the 
pain  in  about  one  quarter  of  the  patients.  In  about  one  percent  of 
the  cases,  it  incorrectly  suggested  that  a  person  suffering  from  MI 
had  some  other  problem.  In  13%  of  the  cases,  it  suggested 
inaccurately  that  a  person  was  suffering  from  cardiac  chest  pain. 
The  magnitude  of  these  errors  is  similar  for  physicians  and 
underlines  the  need  for  the  user  to  make  an  independent  and  careful 
evaluation  of  the  advice  provided  by  the  program. 

Like  the  emergency  room  physicians,  independent  duty  hospital 
corpsmen  have  much  more  information  available  to  them  in  treating 
an  individual  case  than  is  collected  by  the  computer  program.  The 
program  gathers  only  47  responses;  most  of  them  are  either  "yes”  or 
"no.”  The  corpsman  at  sea  generally  has  a  detailed  knowledge  of 
the  patient,  his  past  history  and  behavior,  and  an  acute  awareness 
of  the  degree  that  the  patient's  present  condition  deviates  from 
the  norm.  In  every  case,  the  corpsman,  particularly  when  aided  by 
additional  information  as  is  provided  by  the  computer  program, 
might  be  expected  to  be  a  more  capable  diagnostician  than  the 
computer  program  by  itself.  The  corpsman 's  insight  and  judgment 
remain  indispensable  at  sea. 

A  final  point  should  be  stressed.  Rogers,  Ryack,  and  Moeller 
(1979)  pointed  out  that  the  usefulness  of  these  programs  depends 
not  simply  on  their  being  reasonably  accurate  but  on  being  more 
accurate  than  the  diagnostician  for  whom  the  program  is  intended. 
Since  the  purpose  of  the  computer  program  is  to  serve  as  a 
diagnostic  aid  to  the  independent  duty  submarine  corpsmen,  its 
performance  should  be  compared  with  that  of  a  corpsmen  practicing 
at  sea.  Information  bearing  on  the  accuracy  of  corpsmen  diagnoses 
is  being  collected.  The  present  comparison  with  physicians  serves 
only  to  illustrate  that  the  program  is  not  wildly  inaccurate  when 
compared  with  a  generally  acceptable  standard. 


Work  is  in  progress  to  adapt  Goldman's  program  to 
isolated  Navy  situations. 
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